1. Field of the Invention
The invention relates generally to a system for selective retrieval of video programming from a media library. In particular, the invention relates to a context driven automatic video indexer and a natural language user directed searcher and brower.
2. Description of Related Art
Recent technical advances are enabling a new class of consumer applications involving browsing and retrieval of full broadcast-quality video. Cable modems are bringing megabit bandwidth to the home. Set top boxes include low cost MPEG2 decoders and can also include an HTTP client for web browsing. Disk storage technology is riding a Moore's law curve and is currently at a dollar-per-megabyte point that makes large digital video archives practical. Selective delivery of digital video over IP (an Internet Protocol network) is less well established than these other technologies, but rapid progress is being made in this area. Systems that build upon these component technologies in novel ways can bring new services that meet consumers needs.
Video indexing is one of the areas in which further technology development is needed. To build systems for video database retrieval, we need standard storage and communication protocols at several levels for handling the video program attributes, key frames, associated text streams such as the closed caption, in addition to the usual issues associated with storage and delivery of the bulk media. Video program attributes include such things as the program title, copyright, etc. Currently international standard bodies are active in this area. In particular MPEG7 aims to address this as a natural evolution of the MPEG video standards of the past. For example, see MPEG-7 Context and Objectives, Requirements Group, ISO/IEC JTC1/SC29/WG11 (found at http://drogo.cselt.it/mpeg/standards/mpeg-7/mpeg-7.htm), incorporated herein by reference. The IETF is also working on standards focused in the areas where television and the Internet intersect. For example, see Philipp Hoschka, W3C TVWeb Interest Group, (also found at http://www.w3.org/TV/TVWeb/), Warner ten Kate, “TV URI Schemes—Requirements”, Nov. 10, 1998 (also found at http://www.w3.org/TV/TVWeb/TVWeb-URI-Requirements-19981110), and Warner ten Kate, G. Thomas, C. Finseth, “Requirements TV Broadcast URI Schemes”, Internet Draft, November, 1998 (also found at http://searchietf.org/interent-draft-tenkate-tvweb-uri-reqs-00.txt), all incorporated herein by reference. Meanwhile the industry is moving ahead with implementations that may become de-facto standards in their own right. For example, at the system level we have Microsoft ASF (see Eric Fleishman, “Advanced Streaming Format (ASF) Specification,” Internet Draft, Feb. 25, 1998, also found at http://www.micro soft.com/as f/resources/draft-ietf-fleischman-asf-01.txt, incorporated herein by reference) and Real Networks (see Real Networks, found at http://www.real.com, incorporated herein by reference). At the application level there are Virage and 'SUP and others. For example, see Virage, found at http://www.virage.com, Informedia, found at http://www.islip.com, FasTV, found at http://www.fastv.com, Magnifi, Inc., found at http://www.magnifi.com, and Exclaibur, found at http://www.excalibur.com, all incorporated herein by reference.
When these indexing components become stable, we will have all the building blocks necessary to create systems for browsing video databases that have been manually created. The major broadcasters will likely begin to generate indexes of each video program as part of the normal production process. However, for a significant time, smaller broadcasters will not have the necessary resources to do this. Further, it will be too costly to manually index the many large (several hundred thousand hour) video news archives that exist.
An automated video indexing system is required for these applications. Several such systems have been proposed (for example, at CMU and AT&T. For example, see H. Wactlar, Informedia, Researching Digital Video Libraries, 1998 (also found at http://www.informedia.cs.cmu.edu, and B. Shahraray and D. Gibbon, Automated Authoring of Hypermedia Documents of Video Programs, Proc. Third Int. Conf. On Multimedia (ACM Multimedia '95), San Francisco, Calif., November 1995, both incorporated herein by reference. A common aspect of the successful systems is true multimedia processing in which state of the art techniques from several disciplines are employed. In fact, in many cases it is necessary to extend existing methods in the process of applying them to the domain of video indexing.
The display device itself imposes an additional challenge for broadcast-quality video indexing systems intended for consumers. Typical home PC's are not well designed for the display of interlaced video, and the comparatively low resolution of television sets is not capable of rendering the browsing and retrieval user interfaces (UIs) of the aforementioned systems. We will focus on the home television as the display device. Much of the prior work assumes a desktop environment in which high-resolution graphics are available for the UI, and “postage stamp” quality video is sufficient. In these systems QCIF resolution video serves the purpose of identifying a particular video clip I from a database, and for verifying that it is relevant to a given query. This usually is not sufficient for actually watching the video in the way to which a television viewer is accustomed.
There are several issues to be addressed when designing user interfaces for television-resolution applications. Even though the addressable resolution is approximately 640×480 for NTSC displays, the usable resolution is further reduced by overscan and interlacing. Fonts must be large, and preferably smoothed to avoid interlacing artifacts. Also, usage paradigms dictate that scrollbars are to be avoided. See WebTV Design Guide, WebTV Networks, 1998, as may be found at http://developer.webtv.net/docs/designguide, incorporated herein by reference. Prior systems employ high-resolution UIS to access low-resolution video. We are concerned here with low-resolution UIs for accessing high-resolution video.